A Systematic Review of Fault Prediction Performance in Software Engineering

BACKGROUND – The accurate prediction of where faults are likely to occur in code is important since it can help direct test effort, reduce costs and improve the quality of software. As a consequence, many different fault prediction models have been developed and reported in the literature. However, there is no consensus on what constitutes effective fault prediction. OBJECTIVE – To investigate how the context of models, the independent variables used and the modelling techniques applied influence the performance of fault prediction models. METHOD – A systematic literature review identifying 208 fault prediction studies published from January 2000 to December 2010. A synthesis of the quantitative and qualitative results of those 35 studies which report sufficient contextual and methodological information according the criteria we develop and apply. This synthesis includes a detailed analysis of the relative predictive performances of 203 models (or model variants) reported in 18 of these 35 studies which allow us to calculate precision, recall and f-measure. RESULTS –There are large variations in the performance of these 203 models. The models that perform well tend to be based on simple modelling techniques such as Naïve Bayes or Logistic Regression. Combinations of independent variables have been used by models that perform well. These combinations include product, process and people metrics. Feature selection has been applied to these combinations by models which are performing particularly well. In addition, such models tend to have been trained on large data sets which are rich with faults. CONCLUSION – The methodology used to build models seems to be influential to predictive performance. Although there are a set of fault prediction studies in which confidence is possible, many open questions remain about effective fault prediction. More studies are needed that use a reliable methodology and which report their context, methodology and performance comprehensively. This would enable a meta-analysis across more studies. It would also produce models more likely to be used by industry.